A student scores 74 on Tuesday's test. The pass mark is 75. She fails, misses a place on a programme, and her teacher feels the grade was fair because the number is the number. But suppose she had sat a different version of the same test on Wednesday. Would she have scored 74 again? Probably not: perhaps 71, perhaps 78. The first score was a single draw from a spread of scores she might have got.
Almost everyone treats test scores as exact. People who have studied measurement treat them as estimates with a width. That change of habit shows up everywhere, from exam boards to school league tables, and from hiring tests to the AI benchmarks of the past few years. This piece is about the width, the question of whether the test measures the right thing at all, and what happens when a number gets high stakes attached.
The oldest model in measurement, classical test theory, goes back to Charles Spearman's work in 1904 and was set out fully by Frederic Lord and Melvin Novick in 1968. It says an observed score is the sum of two parts: a true score, the average you would get over endless equivalent versions of the test, and a random error. Errors come from the particular questions chosen, the student's mood and sleep, a lucky guess, a marker's mood.
Reliability, written r, is a number from 0 to 1: the share of the differences between students that reflects real differences and not noise. A test with reliability 0.85 and a spread (standard deviation) of 10 points has an SEM of about 3.9 points, so about 95 times in 100 a student's observed score falls within roughly ±7.6 points of her true score. A pass mark inside that band decides some students by luck.
A student has a true score of 70. The test has a spread of 10 points and the pass mark is 75. Set the reliability, then sit the test again.
The simulated scores use a fixed pseudo-random sequence, so they repeat every time you reload. The chance of passing is computed from the normal distribution.
Reliability can be estimated several ways. Test–retest compares scores on two occasions. Parallel forms compares two versions. Internal consistency, most often Cronbach's alpha (1951), asks whether the items hang together. For essays, inter-rater reliability asks whether two markers agree; it is often much lower than people assume until markers are trained and use a scoring guide.
The practical rule is the Spearman–Brown formula (1910): lengthening a test with items of the same quality raises its reliability. If a test with reliability r is made k times longer, the new reliability is k × r ÷ (1 + (k − 1) × r). Starting at 0.60, doubling gives 0.75, and a six-fold length gets to 0.90. The gains shrink as the test grows, which is why a 200-item exam is not twice as good as a 100-item one. One more fact: reliability caps validity. A noisy test cannot track anything well, though a steady test can still track the wrong thing.
A 10-item test has reliability 0.60 (invented). Change the number of similar items.
A bathroom scale that is always 2 kilograms off is reliable and not valid. Validity is whether the scores support the interpretation you want to draw from them. Samuel Messick (1989) argued that validity is not a property of a test but of the use of its scores, built from evidence about content, response processes, internal structure, relation to other measures and consequences. Michael Kane's argument-based approach (2006) asks for an explicit chain from the task to the claim and a test of its weakest link.
Two threats do most damage. Construct underrepresentation is a test too narrow for what you claim: a "writing" score from a grammar quiz. Construct-irrelevant variance is something else moving the score: a maths word-problem test that a pupil fails because the English is hard. Both can leave reliability high. The classic old example of reliable and invalid is measuring skull size to the millimetre to estimate intelligence.
Classical theory treats a test as a whole. Item response theory (IRT) models each question. Georg Rasch (1960) and Frederic Lord worked out that the chance of a correct answer can be written as a curve against the student's ability. Each item has a difficulty (b: the ability at which you have even odds) and a discrimination (a: how sharply the curve rises there). In the two-parameter model, P(correct) = 1 ÷ (1 + e−a(θ − b)), where θ is ability.
The payoff is the adaptive test. Pick the next item near your current estimate of ability: a question far too easy or too hard tells you little. After about a dozen well-chosen items the estimate can be as good as a much longer fixed test. Computer-adaptive testing has been used for professional and admissions tests such as the GMAT since the 1990s. A guard on the other side is differential item functioning: an item that behaves differently for two groups of equal ability is flagged for review.
All items in the adaptive run have a = 1.2 and a bank of difficulties from −3 to 3 in steps of 0.5. Ability is estimated after each answer as the mean of the plausible values on a grid, starting from a standard normal prior. Answers are random draws, so each run differs.
Robert Glaser (1963) separated two meanings of a grade. A criterion-referenced grade says what you can do: an A means you met the standard, and everyone can get one. A norm-referenced grade says where you stand: an A goes to, say, the top fifth, however well everyone did. Grading "on a curve" is the second. It fits selection (choose the best twenty) and fits poorly with teaching, since a strong class still produces failures and a weak one produces undeserved As.
The sharpest recent lesson came in England in 2020. With exams cancelled in the pandemic, the regulator Ofqual used a statistical model that adjusted teachers' predicted grades using each school's past results, to hold national results steady. About 39 percent of A-level grades came out lower than the teachers' predictions, small classes were treated less fairly, and after protests the government and Ofqual reverted to teacher-assessed grades on 17 August 2020. The model delivered a stable national distribution and an unfair result for individuals, which is what a norm-style fix does when the aim shifts.
Grading scales differ around the world and are not directly comparable: Germany uses 1 to 6 with 1 best, Finland's schools use 4 to 10, Mexico 0 to 10 with 6 as a pass, and many US schools use letters.
A summative assessment sums up learning at the end: a final exam. A formative one is used while learning is under way to change teaching. Paul Black and Dylan Wiliam's 1998 review, with its short companion Inside the Black Box, argued that classroom assessment used this way produced substantial learning gains. Later reviewers, such as Randy Bennett (2011), have said the evidence was weaker and less specific than the headlines. A kind of agreement remains: the useful thing is the loop of finding out what students understand and doing something about it, not any single technique.
Feedback is the part to handle with care. Avraham Kluger and Angelo DeNisi's 1996 meta-analysis found that, on average, feedback helped, but more than a third of the interventions made performance worse, especially when feedback drew attention to the person and not the task. John Hattie and Helen Timperley (2007) sorted feedback by whether it addressed the task, the process, self-regulation or the self. The Education Endowment Foundation's Toolkit rates feedback as a high-impact approach on average, with wide variation. Low-stakes quizzes join this and Learning Sciences (36/53): retrieval is both a measurement and a way of learning.
International studies turn school results into rankings. PISA (OECD) tests 15-year-olds every three years; the 2022 results, published in December 2023, covered about 690,000 students in 81 countries and economies. The OECD average in mathematics fell to 472 from 489 in 2018, a 15-point drop, the largest between two cycles, and Singapore led with 575. TIMSS (IEA) tests grades 4 and 8; in its 2023 cycle Singapore led grade 8 mathematics with 605, ahead of Chinese Taipei (602), Korea (596) and Japan (595). The pandemic is one likely contributor to the declines, and researchers disagree about how much.
A published rank hides the error. Each country's score is an estimate from a sample of students and questions with a standard error, and two countries a few points apart may not differ. Comparisons across countries add further problems: different populations of 15-year-olds in school, translation of items, and motivation on a test that counts for nothing.
Then there are the stakes. Donald Campbell stated in 1976 that the more a quantitative indicator is used for decisions, the more it is subject to corruption pressures and the more it distorts what it measures; Charles Goodhart made a similar point about monetary targets. In education that looks like teaching to the test, narrowing the curriculum to what is examined, coaching, and sometimes cheating. Systems respond differently. High-stakes selection exams are central in China (the gaokao, taken by about 13 million candidates in recent years), South Korea (the CSAT) and India (JEE and NEET); France has the baccalauréat; Finland has had little standardised testing until the upper-secondary matriculation exam. Defenders of high-stakes testing argue that it gives objective, comparable and fair criteria; critics argue that it distorts teaching and rewards test-taking skill. Evidence and values both come in, and neither settles it alone.
A test score is an estimate of something hidden, with an error you can calculate and a validity you must argue for. Steady tests need enough good items, adaptive tests choose them well, grades need a stated standard, feedback is useful when it changes what happens next, and any number given high stakes will be bent toward.
Try this: next time you see a ranking or a test score, ask three questions. How wide is its error? Does it measure what people say it does? What would someone do to raise it?
Sources. Spearman 1904 and 1910; Lord and Novick, Statistical Theories of Mental Test Scores (1968); Cronbach, Psychometrika 1951; Messick, in Educational Measurement (3rd ed., 1989); Kane, in Educational Measurement (4th ed., 2006); AERA, APA and NCME, Standards for Educational and Psychological Testing (2014); Rasch 1960; Lord, Applications of Item Response Theory (1980); Glaser, American Psychologist 1963; Black and Wiliam, Assessment in Education 1998; Bennett, Assessment in Education 2011; Kluger and DeNisi, Psychological Bulletin 1996; Hattie and Timperley, Review of Educational Research 2007; Campbell, 1976 (Western Michigan University occasional paper); Goodhart 1975; OECD, PISA 2022 Results (December 2023); IEA, TIMSS 2023; Ofqual and press reports on the August 2020 A-level grading (17 August 2020 reversal); Education Endowment Foundation Toolkit. Every computed number in the interactives was checked in node. The classes, countries and scores in the toys are invented.